1. Field of the Invention
This invention relates to apparatus and methods for dynamically switching between transfer-ready (XFR_RDY) enabled mode and transfer-ready (XFR_RDY) disabled mode.
2. Background of the Invention
During fibre channel protocol (FCP) write operations, XFR_RDY may be used by the FCP target device to notify the FCP initiator that the target device is ready to receive a burst of FCP data. The use of XFR_RDY for the first burst of data is negotiated between two FCP ports at process login (PRLI) time. FCP ports negotiate to either use XFR_RDY on the first burst of FCP data during write operations, or to disable XFR_RDY on the first burst. If XFR_RDY is enabled, then the target device, after receiving an FCP write command, will allocate space for the first burst of FCP data before sending a XFR_RDY message to the initiator. Likewise, the initiator may only send a burst of FCP write data after it has received a XFR_RDY message. If XFR_RDY is disabled, then the initiator assumes that the target device always has x bytes of data available to receive the first burst of data, and the initiator will send write commands immediately followed by FCP data, without waiting for a XFR_RDY message. Operating without XFR_RDY enabled is especially beneficial for applications that are latency bound, such as data replication over long distances.
Modern fibre channel ports typically support at least 4K of concurrent fibre channel exchanges, and at least 64K for each burst of data. Assuming these values, an FCP device that supports XFR_RDY disabled must have at least 64K×4K=256 MB of memory per port available to receive the first burst of data. This is a large amount of memory, especially considering that modern storage controllers may support upwards of 128 ports, which translates into 32 GB (256 MB×128 ports) of memory reserved for first bursts of data. Not all storage controllers have the resources available to reserve this much memory to receive first bursts of data with XFR_RDY disabled. Without guaranteeing enough buffer space available to receive the first bursts of data, a storage controller that needs to be able to operate with XFR_RDY disabled has the following options:
The first option is to not support operation in XFR_RDY disabled mode. This is the option implemented by most fibre channel devices. However, this option is not viable in many configurations, such as configurations that replicate data over long distances, since it unacceptably impacts write performance. This is due to the fact that disabling XFR_RDY requires the initiator to wait for a XFR_RDY message from the target before a data transfer can occur. Such a delay is unacceptable in many configurations.
The second option is to support a limited number of exchanges between an initiator and target. This option is not viable in competitive environments where oversubscription is common. Most controllers need to support a larger number of initiators logged in per port, with each initiator sending a large number of exchanges concurrently.
The third option is to support operation in XFR_RDY disabled mode, but to limit the size of the first burst of data. This option is not viable for storage controllers with cache track-based architectures, which may be configured to send an XFR_RDY message for each 64K track. There is therefore a need to have a burst size that is able to support a minimum of 64K.
The fourth option is to support operation in XFR_RDY disabled mode, but not guarantee that there are enough buffers to accommodate the maximum number of concurrent write transfers that may occur with XFR_RDY disabled. In other words, the target device is configured to allow the oversubscription of the buffer space available. This option relies on the fact that, most of the time, the workload will not require all resources. However, if the workload is high enough, and more buffer space is required than is physically available, the target controller must either discard or abort exchanges. This leads to I/O errors, and results in performance degradation and possible SAN congestion with its various consequences, including impacting other devices in the SAN.
The fifth option is for the FCP ports to negotiate the disabling of XFR_RDY during process login. When buffer space falls below a certain threshold, the target is configured to drop new incoming exchanges and/or log out the host. This may force the host to log in again, at which time the FCP ports can renegotiate to enable XFR_RDY. This option requires terminating all I/O before XFR_RDY usage can be enabled or disabled, causing undesirable performance degradation.
In view of the foregoing, what is needed is a solution that allows a device to take advantage of the performance gains possible with XFR_RDY disabled, while not requiring that a maximum supported number of FCP write operations can run concurrently. Further needed is a solution that can efficiently enable and disable XFR_RDY with minimal write performance degradation. Ideally, such a solution would allow a device to enable and disable XFR_RDY as workloads vary, and on an exchange-to-exchange basis.